Experiments on the zero frequency problem

The best algorithms for lossless compression of text are those which adapt to the text being compressed [1]. Two classes of such adaptive techniques are commonly used. One class matches the text against a dictionary of strings seen and transforms the text into a list of indices into the dictionary. These techniques are usually formulated as a variant on Ziv-Lempel (LZ) compression. While LZ compressors do not give the best compression they are widely used because of their simplicity and low execution overhead. The best compression is obtained by another class of compressors which use adaptive statistical modelling. These split compression into two steps. The rst step accumulates a statistical model of the characters seen so far in the input text. As each character is encoded this model is used to generate a probability distribution over those characters which can occur next. Arithmetic coding is then used to optimally encode the character which actually does occur with respect to this distribution. The best compression has been obtained from a series of variants of PPM modelling [1]. PPM models are built up by counting the characters that have occurred following contexts of prior characters. For example, all the characters following `a' are recorded. The next time `a' occurs the counts associated with it are used to generate the probability distribution for the following character. The PPM techniques blend together the predictions from contexts of varying lengths to arrive at an overall probability distribution. For practical reasons of memory usage and execution time most PPM variants x an upper bound to the lengths of the contexts, although recently a variant which uses unbounded length contexts has been very successful [2]. The focus of this paper is the problem of transforming the set of counts accumulated for a particular context into a probability distribution. To simplify our discussion and later experiments we will focus on the case when the alphabet of characters is binary with just two symbols: 0 and 1. Now in a statistical model each context will deliver two counts: C0, the number of times a 0 has occurred, and C1, the number of times a 1 has occurred. A naive estimate of the probability of character i could be obtained by the ratio